Image handling and display in X-ray mammography and tomosynthesis

ABSTRACT

A method and system for acquiring, processing, storing, and displaying x-ray mammograms Mp tomosynthesis images Tr representative of breast slices, and x-ray tomosynthesis projection images Tp taken at different angles to a breast, where the Tr images are reconstructed from Tp images

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/646,317, now U.S. Pat. No. 9,042,612, filed Oct. 5, 2012; which is acontinuation of U.S. patent application Ser. No. 13/027,771, filed Feb.15, 2011, now U.S. Pat. No. 8,285,020; which is a continuation of U.S.patent application Ser. No. 12/539,460, filed Aug. 11, 2009, now U.S.Pat. No. 7,916,915; which is a continuation of U.S. patent applicationSer. No. 11/827,909, filed Jul. 13, 2007, now U.S. Pat. No. 7,616,801;which is a continuation of U.S. patent application Ser. No. 11/604,069,filed Nov. 24, 2006, now abandoned; which is a continuation-in-part ofU.S. patent application Ser. No. 11/271,050, filed Nov. 11, 2005, nowU.S. Pat. No. 7,577,282; which claims the benefit of provisionalapplications Nos. 60/628,516and 60/631,296filed respectively on Nov. 15and 26, 2004, and incorporates herein by reference the entire contentsof each of said earlier-filed patent applications. U.S. patentapplication Ser. No. 11/604,069, filed Nov. 24, 2006, now abandoned, isalso a continuation-in-part of U.S. patent application Ser. No.10/723,486, filed Nov. 26, 2003, now U.S. Pat. No. 7,831,296; and isalso a continuation-in-part of U.S. patent application Ser. No.10/305,480, filed Nov. 27, 2002, now U.S. Pat. No. 7,123,684; thedisclosures of which are incorporated herein by reference.

FIELD

This patent specification pertains to x-ray mammography andtomosynthesis, and more specifically to techniques and equipment foracquiring, processing, storing and displaying mammograms, tomosynthesisprojection images, and tomosynthesis reconstructed images, and tomedical image softcopy reading systems, to hanging protocols and toother medical image display features.

BACKGROUND AND SUMMARY

Mammography has long been used to screen for breast cancer and otherabnormalities and for diagnostics. Traditionally, mammograms were formedon X-ray film, but more recently flat panel digital imagers have beenintroduced that acquire a mammogram in digital form and therebyfacilitate analysis and storage and provide other benefits as well.Further, X-ray tomosynthesis of the breast has been proposed recently,as discussed in the earlier-filed applications identified above, andclinical testing has been carried out. The assignee of this patentspecification, Hologic, Inc., has demonstrated at trade shows in thiscountry a fused, multimode mammography/tomosynthesis system that takeseither or both types of images, and either while the breast remainsimmobilized or in different compressions of the breast. Dedicated breasttomosynthesis systems also have been proposed.

Tomosynthesis as used in the systems and methods disclosed in thispatent specification typically involves acquiring a plurality oftomosynthesis projection images Tp at respective angles relative to thebreast, and reconstructing therefrom a plurality of tomosynthesisreconstructed images Tr representative of breast slices that haveselective thicknesses. Proper display techniques are desirable to makethe presentation of Tp and/or Tr images (collectively referred to hereas T images) more effective and efficient for review by healthprofessionals. When tomosynthesis projection images Tp are acquiredalong with conventional 2D mammograms Mp, improved display methods aredesirable that facilitate the display of both T and Mp images. Effectivedisplay approaches also are desirable when tomosynthesis images Tpand/or Tr that are acquired at one time need to be compared tomammograms Mp and/or to tomosynthesis images Tp and/or Tr acquired at adifferent time. Effective displays also are desirable when only Trand/or Tp images are being displayed. Another display issue relates toComputer Aided Detection (CAD) methods that use computer analysis ofimages to identify locations and possibly other characteristics ofsuspected abnormalities. CAD marks currently are placed on or areotherwise associated with mammogram images Mp, but it may be useful toplace them at the appropriate location on Tr and/or Tp images o tootherwise associate them with Tr/Tp images. Conversely, it may bedesirable to obtain CAD marks by processing Tp and/or Tr images, andplace them at appropriate locations on Mp images. Here the notation Mprefers to a conventional mammogram, which is a two-dimensionalprojection image of a breast; the term Mp encompasses both a digitalimage as acquired by a flat panel detector or another imaging device andthe image after conventional processing to prepare it for display to ahealth professional or for storage, e.g. in the PACS system of ahospital or another institution. Tp refers to an image that is similarlytwo-dimensional but is taken at a respective tomosynthesis angle betweenthe breast and the origin of the imaging X-rays (typically the focalspot of an X-ray tube), and also encompasses the image as acquired aswell as the image after being processed for display or for some otheruse. Tr refers to an image that is reconstructed from images Tp, forexample in the manner described in said earlier-filed patentapplications, and represents a slice of the breast as it would appear ina projection X-ray image of that slice at any desired angle, not only atan angle used for Tp or Mp images. The terms Tp, Tr and Mp alsoencompasses information, in whatever form, that is sufficient todescribe such an image for display, further processing, or storage. Theimages Mp, Tp and Tr typically are in digital form before beingdisplayed, and are defined by information identifying properties of eachpixel in a two-dimensional array of pixels. The pixel values typicallyrelate to respective measured or estimated or computed responses toX-rays of corresponding volumes in the breast (voxels or columns oftissue).

Yet another issue concerns the large storage requirements oftomosynthesis images Tp and/or Tr. Because the reconstructed datasetsfor Tr images are large, it may be better in some circumstances to storeunreconstructed projections images Tp, which require less storage.Transmission times to the storage device, and from the storage device tothe display workstation, can thus be reduced. The Tp images in this casecan be reconstructed to Tr images just prior to viewing that requires adisplay of Tr images. Further, it may be desirable that images viewed ona workstation are the same or at least comparable to images viewed on adifferent workstation, or the same or at least comparable to previouslyviewed images of the same dataset, even if the software and/or hardwareof the workstation or acquisition system or intermediate storage orprocessing systems, have changed.

Yet another issue concerns the processing time required to reconstructtomosynthesis images Tr. Because of relatively long reconstructiontimes, one possible approach is to perform reconstructions at anacquisition console, and send the already-reconstructed images todisplay workstations. This can allow a greater reading throughput ifthere are several acquisition systems that are all pushing images to oneor more display workstations. The system can be designed so that it canhandle M acquisition consoles sending their images to all N displayworkstations.

In one non-limiting example disclosed in this patent specification,acquisition and display of x-ray images starts with acquiring x-raymammography image data representative of projection mammography imagesMp of patients' breasts and x-ray tomosynthesis image datarepresentative of projection images Tp taken at different angles of atleast a source of imaging x-rays relative to the patients' breasts(e.g., different angles of the focal spot in an X-ray tube relative animmobilized breast). This acquisition can be performed by a single unit,using a single X-ray tube and a single flat panel digital imager or someother imaging device, configured to selectively acquire one or both ofthe mammography and tomosynthesis image data, in the same compression ofa patient's breast or in different compressions, and at the same imagingsession or at different times. The disclosed system and method use atleast a subset of the acquired Tp images to form reconstructedtomosynthesis images Tr representative of slices of the breasts thathave selected orientations and thicknesses. The system and methoddisplay at least a selected subcombination of the Mp, Tr and Tp images,preferably for concurrent viewing or at least for viewing in a singlesession, and preferably but not necessarily while showing, at or nearthe displayed images, respective labeling symbols identifying them asMp, Tr or Tp images and possible other information that facilitatesdetection/diagnosis of abnormalities, such as information indicative ofthe position and orientation of the slices represented by Tr images, thethicknesses of such slices, etc. The information can be in alphanumericor non-numeric form such as in the form of graphics and/or icons.

The method and system can further generate or otherwise obtain computeraided detection (CAD) marks for suspected abnormalities in said Mpimages, and can display said marks at corresponding locations on or inassociation with Tr and/or Tp images that are related, e.g. byorientation or otherwise, with respective Mp images. In addition tolocation information, the CAD marks can provide information regarding,for example, the type of suspected abnormality and/or a confidence levelthat a mark points to an actual abnormality. CAD marks that areinitially generated from or are otherwise related to some of the Tr, Tpor Mp images can be displayed at or in association with images fromwhich they were not generated or with which they were not initiallyassociated, at corresponding or at least related locations. Tp imagescan be stored together with version information indicative of at leastone of an acquisition configuration used to acquire them and areconstruction configuration used to reconstruct Tr images from said Tpimages, to thereby enable later reconstruction of Tr images that matchthose reconstructed at an earlier time. Alternatively, or in addition,Tp images can be stored together with version information related towhen they were acquired and can be later reconstructed into Tr imagesusing a reconstruction configuration that matches the versioninformation. A reconstruction configuration can be provided that has atleast two different versions of reconstruction software, so that Trimages can be reconstructed using a version of the reconstructionsoftware that matches the version information of the Tp images orearlier Tr images. Tr images can be reconstructed from only a subset ofthe acquired Tp images, which in an extreme case means reconstructionfrom a single Tp image to yield a Tr image that is equivalent of the Tpimage. Tr images representative of at least two breast slices thatdiffer in thickness can be formed, for example using MIP (MaximumIntensity Projection) methods or a summing method that may or may notuse different weighting of the summed pixel data. The display can betoggled between Tr images representative of breast slices havingdifferent thicknesses, wherein the slices may or may not overlap inspace. Through computer-processing, the volume of a lesion can becomputed and displayed from information contained in the Mp, Tr and/orTp images. The display can show concurrently, for example, Tr imagesreconstructed from a current acquisition of Tp images and at least oneMp image obtained from a previous acquisition involving a differentbreast compression. The concurrent display can be on the same ordifferent display monitors, and can include at least Mp and/or Trimages, or at least Mp and/or Tp, images, or at least Tr and/or Tpimages, or all three types of images, and can instead or additionallyinclude 3D images formed from some or all of the acquired X-ray data,image data and/or from Mp, Tr and/or Tp images. Information indicativeof status of loading Tr images for display can be shown as a part of thedisplay. Different images can be displayed at different pixel sizes orfields of view or, alternatively, they can be selectively equalized bypixel size or field of view by selected types of interpolation orextrapolation, for example by up-converting to a smaller pixel size andthus a higher converted pixel count or by down-converting to a largerpixel size and thus a lower pixel count.

The display of Mp and T images such as Tr images can include displayingnon-numeric indications of various properties of the displayed images.Such non-numeric indications can include indications of respectivelevels, spacing and slice thickness for displayed Tr images relative toa compressed breast imaged in an Mp image or relative to some otherframe of reference, for example in the form of cross-lines on a barrelated to the breast as displayed as an Mp image, wherein the height ofthe bar may relate to the thickness of the compressed breast, and/ornon-numeric indications of respective thicknesses of breast slicesrepresented by displayed Tr images, for example in the form ofcross-bars of respective thickness on a bar related to Mp images, and/ornon-numeric indications of the inclination angle of the slice or slicesrepresented by one or more Tr images relative to a selected frame ofreference such as the compressed breast that was imaged to generate thedata from which the Tr images were reconstructed, and/or non-numericindications of other parameters. Instead, or in addition to suchnon-numeric indications, numerical indications can be provided anddisplayed of the position of a slice image Tr relative to, e.g., abreast imaged in an image Mp, the thickness of the slice represented ina Tr image, and/or the orientation of that slice. Mp and T images suchas Tr images can be shown overlaid on each other, and toggling can beallowed to switch between the image that is visible at the time and animage that is not, i.e. toggling between the 2D Mp image and the 3D Trimage or images. Similar toggling is available between different typesof T images, of between different T images of the same type. Inaddition, other image display effects can be provided, such as, withoutlimitation, fade-in/fade-out and blending two or more images atrespective weightings, image overlays and masking, or other effects, ascommonly used in post-production of television images and in known imageprocessing software such as Photoshop from Adobe.

T images such as Tr and/or Tp images can be displayed in cine mode, withselective control over the speed of changing from one image to another,the order of images for display relative to an order in which they werereconstructed or acquired, the selection of the first and last images inthe cine sequence, and/or other parameters. For example, at least twosets of Tr images, each set reconstructed from different acquisitions ofTp images, can be shown concurrently and scrolled through and/ordisplayed in cine mode in synchronism. For example, the two sets of Trimages can be synchronized such that the first and last slice images ofeach set can appear on the display at the same time, which may beimplemented in different ways, such as by making the number of imagesthe same (but possibly representing slices of one thickness in one setand slices of a different thickness in the other set), or by scrollingthrough images of one set faster than through images of the other set,or in some other way. Making the number of Tr images of one set matchthat of the other set may involve interpolating images of new slices toeither reduce the number of images/slices in one set or increase it. Theimages in that case in one set of Tr images can be made to representslices that are all the same thickness or slices that include some thatare thicker or thinner than other slices.

Tr images can be displayed in scroll or cine mode in different orders,such as by starting with the image representing the bottom of the breast(the breast side resting on a breast platform during imaging) toward toimage representing the top of the breast (the breast side compressed bya compression paddle), or in reverse order, or by starting with a Trimage representing a selected intermediate slice and proceeding towardthe top or the bottom of the breast, or in some other desired order. Theinitial image that is displayed can be the bottom image, the top image,or a selected intermediate image. In a scroll or cine mode, the displaycan show every image of a Tr set for a breast, or a selected subset,such as every other image or some other subset of all Tr imagesrepresenting a breast. The display system can have a default mode for anew user in which it starts the scroll or cine display with the bottomimage, but with provisions for this default to be changed to anotherdisplay protocol for that user in which the starting image is anotherone of the Tr images. A selection of initial or default display modescan be provided relating to the order, speed, slice thickness and/orother parameters of display of images, and user selection among thosemodes can be allowed. Similar procedures are available in the case whenthe orientation of the Tr slices is not the same as that of the Mp or Tpslices.

Thus, information regarding image data acquisition, storage,reconstruction and/or other parameters can be selectively displayed. Forexample, Tr images can be identified on the display as such, thusproviding an image type indication display, and additionalidentification can be displayed in association with a displayed imagethat identifies the position in the breast of the slice represented bythe displayed Tr image, the thickness of the slice, the orientation ofthe slice, and/or some other property associated with the displayed Trimage. Corresponding or at least similar display of information can beassociated with different types of displayed images, such as Tp and Mpimages and displayed in association with the display of the images. Inthe alternative, images such as Tr, Tp, and/or Mp can be displayed indifferent combinations and/or sub-combinations that may include the sametype of images displayed concurrently or toggled, or different types ofimages displayed concurrently or toggled, without providing some or anyof the identifications discussed above. Tr images can be printed in anN×M format (where N and M are positive integers), and printing of anyimages displayed on one or more monitors in WISIWIG format can beprovided.

Compression of Mp, Tp, and/or Tr images and/or of other image data canbe selectively carried out prior to storing or archiving the images. Thecompression can be lossless, or it can be lossy to a selected degree.Reconstruction of Tr images can be selectively carried out fromcompressed Tp images, preferable after suitable decompression.Window/level controls can be provided for at least selected ones of thedisplayed Tr images, and the controls can be set by the user, orautomatically, to control the window width and/or the window level ofonly one, or only selected ones, or all of the displayed images. Imageregions can be magnified for display, and the window/level controlsautomatically applied to the magnified regions. The Tr, or the Tp, orboth the Tr and Tp images, can be stored in PACS storage, and can beassociated with related Mp images and/or with selected CAD information.The Tp images can be acquired by using coarser binning, e.g. in adirection of relative motion between the source of imaging x-rays and abreast during image acquisition, or in both directions. Alternatively,such binning can be done after the Tp images are acquired, to therebyreduce storage and further processing requirements. The Mp, Tr, and/orTp images that are concurrently displayed can be based on image dataacquired from the same breast of a patient while the breast remainsimmobilized under compression that can remain the same or change betweenthe acquisition of Mp images and Tp images. Alternatively, the Mp and Tpimages can come from different acquisitions at different times ordifferent breast compressions. Image data for Tp images acquired at twoor more acquisition units can be supplied to and reconstructed into Trimages at a single reconstruction unit, from which one or more datadisplay units can acquire Tr, Tp, and/or Mp images for display, or imagedata for Tp images can be stored as such and only reconstructed into Trimages immediately prior to display thereof. Images with differencecharacteristics such as pixel size, brightness, gamma curves, etc. canbe processed to make selected ones of their characteristics sufficientlysimilar to facilitate comparison.

An additional or alternative display approach uses the Tp and/or Trimages in stereoscopic display. For example, when any two Tp imagestaken at different angles to the breast are displayed concurrently andviewed such that each is seen by a different eye of the observer, depthinformation is visualized. Similarly, when any two Tr images arereconstructed such that their image planes are at an angle to eachother, depth information can also be perceived.

Images of different types and from different sources can be displayed indesirable size and resolution. For example, an image can be displayed in(1) Fit To View Port mode, in which the size of the displayed image sizeis maximized such that the entire imaged breast tissue is visible, (2)True Size mode, in which a display pixel on the screen corresponds to apixel of the image, or (3) Right Size mode, in which the size of adisplayed image is adjusted so that it matches that of another imagethat is concurrently displayed or with which the displayed image is orcan be toggled. For example, if two images of the same breast are takenand are not the same size or do not have the same special resolution,provisions are made to selectively zoom in or zoom out one of them, orzoom both, such that they appear to be the same size on the screen whenthey are concurrently displayed or the user toggles between them, tofacilitate comparison or to otherwise facilitate detection/diagnosis.Known interpolation/extrapolation and weighting techniques can be usedin such re-sizing, and known image processing technology can be used tomake other characteristics of the displayed images similar in a way thatfacilitates detection/diagnosis.

Selected hanging protocols are provided that are unique to the differenttypes of images with which the disclosed system deals. As one example,the hanging protocols for 2D images (e.g. Mp images) and 3D images (e.g.Tr images) are linked so that when one type of image is displayed for agiven breast the other type is displayed as well. For example, when theMp image of a breast is displayed, a tile of the Tr images and/or of theTp images is automatically displayed at the same time, with a desiredhanging protocol that may involve scrolling or cine mode presentation,or may require user input so select a particular subset of the Tr and/orTp images or a particular individual Tr/Tp image. Thus, a combinedhanging protocol set can be provided for 2D and 3D images that areconcurrently displayed or toggled such that only one type is displayedat one time. In addition, the combined hanging protocol can includeprovisions for linked display of CAD information associated with one orboth of the 2D and 3D images involved in the hanging protocol.Alternatively, the hanging protocols for 2D images are made differentfrom those for 3D images.

When CAD information is available that is associated with any of theimages that can be displayed, linking can be provided such that CADinformation derived from one of the types of displayed images can beautomatically associated with corresponding locations on anotherdisplayed image. For example, if CAD information has been derived on thebasis of an Mp image and the Mp image is displayed at the same time as,or is toggled with, Tr images, provisions are made to selectivelydisplay CAD information associated with the appropriate locations on Trimages. Alternatively, if CAD information is derived on the basis of Trand/or Tp images, it can be automatically selectively displayed inassociation with an Mp image that is displayed at the same time or istoggled with the Tr/Tp images. In one example, an Mp image with CADmarks thereon remain displayed on the screen while Tr images of the samebreast are scrolled or shown in cine mode on the screen, to facilitateidentification and/or assessment of suspected abnormalities in the Trimages.

When Tr images are displayed, provisions arc made to selectively adjustthe thickness of the slice represented by any displayed Tr image. Forexample, if it is desired to display a Tr image of a slice (slab) ofbreast tissue that is 1 cm thick but the available Tr images represent 1mm thick slices, 10 of those Tr images can be combined into a single newTr image that represents the 1 cm slice, using for example a known MIP(maximum intensity projection) technique.

For storage, transmission to remote locations, and/or other purposes,the images can be formatted consistent with DICOM standards. Forexample, each raw or displayed projection image set or reconstructedslice image set for a single view is stored as a single SecondaryCapture image instance according to DICOM. The image pixel data can beencoded in a selected compressed format (CODEC) that includes allprojection or slice images

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating flow of data through a systemwhere reconstruction of tomosynthesis slice images Tr occurs after (or,alternatively, before) storage of acquired tomosynthesis projectionimages Tp.

FIG. 2 is a block diagram illustrating flow of data where thereconstruction of images Tr occurs before storage.

FIG. 3 illustrates an example where four units acquiring Tp images feeda single unit that reconstructs Tr images.

FIG. 4 illustrates an example where each of four units acquiring Tpimages has its own unit for reconstructing Tr images.

FIG. 5 illustrates an example of displaying Tr (or Tp) images andmammogram images Mp at separate areas of a single screen or on differentscreens.

FIG. 6 illustrates an example where an Mp image and a Tr image may beshown at the same or substantially same area on a screen, with anexample of a non-numeric indication of a thickness and position in thebreast of a breast slice represented by a Tr image.

FIG. 7 illustrates a concurrent display of Tr and Mp images, at separateareas on a screen or as combined images.

FIG. 8 illustrates a display of Mp/Tr images with CAD marks and anon-numeric indication of Tr images in which CAD marks exist.

FIG. 9 illustrates stereoscopic display of Tp images.

FIG. 10 illustrates stereoscopic display of Tr images.

FIG. 11 is a block diagram illustrating major elements of amammography/tomosynthesis system.

FIG. 12 illustrates an example of a non-numeric display indicative ofthe angle of a slice represented by a Tp image relative to a frame ofreference such as the breast platform.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In describing preferred embodiments illustrated in the drawings,specific terminology is employed for the sake of clarity. However, thedisclosure of this patent specification is not intended to be limited tothe specific terminology so selected and it is to be understood thateach specific element includes all technical equivalents that operate ina similar manner.

FIG. 1 illustrates flow of data in one example of a system disclosed inthis patent specification. An image data acquisition system 1 acquirestomosynthesis and/or mammography image data for Tp and/or Mp images ofpatients' breasts, and can take the form of and use the acquisitionmethods of any of the systems disclosed in said earlier-filed patentapplications. Following acquisition, the data describing projectionimages Tp are sent to storage device 2, which can include a PictureArchiving and Communication System (PACS) storage, for example of thetype commonly used in hospitals and other healthcare facilities,preferably a DICOM-compliant PACS. When images are needed for display 4,the data for Mp and/or Tp images are sent, from either acquisitionsystem 1 or from storage device 2, to a computer system 3 configured asa reconstruction engine that can perform tomosynthesis reconstructioninto images Tr representing breast slices of selected thickness and atselected orientations, as disclosed in said earlier-filed patentapplications. The reconstructed slice images Tr are then sent to adisplay system 4 so that they can be viewed. If the reconstructionengine 3 is connected to display 4 via a fast link, then large datasetscan be transmitted quickly.

Over time, there will likely be improvements to the acquisition systemsand to the display systems, which may result in hardware and softwareupgrades and changes to reconstruction algorithms. This can createissues in viewing images taken previously. It may be important to beable to recall from storage and reconstruct an image that looksidentical (or is at least comparable) to the way it looked when it wasreconstructed and displayed in the past, or vice versa. Consider theexample where an improvement in reconstruction algorithms improves imagequality so as to allow detection of a cancerous lesion in an image whereit was not visible using a previous version of the reconstructionalgorithm and the then existing standard of care. While it could beuseful to see older images processed with the newer algorithms, it mayalso be important to allow the re-display of images as they were viewedduring an original detection/diagnosis. One way to accomplish this inaccordance with the disclosure in this patent specification is to put aversion number or some other information in the data for Tp images,which identifies the software and/or hardware versions of the Tp imagedata acquisition and/or Tr image reconstruction system at the time ofacquisition, or to otherwise associate such information with the Tpimages. During reconstruction at a later time, the reconstruction enginereads this version number or other similar information and reconstructsusing the appropriate algorithm. Thus, system upgrades can maintain alibrary of older algorithms and/or hardware so as to be able toreconstruct using the proper technique.

An alternative design is illustrated in FIG. 2. In this example, thereconstructions at unit 3 occur near or at the acquisition station 1,and it is the reconstructions Tr that are sent to storage system 2 anddisplay devices 4. One advantage of the configuration of FIG. 2 is inthe way it handles acquisition upgrades—if a new hardware/softwareversion has a modified reconstruction algorithm, then all Tr imagesreconstructed from Tp image data taken after the upgrade willautomatically reflect this new algorithm, and Tr images reconstructedfrom Tp image data taken prior to the upgrade will have beenreconstructed with the older version and properly stored as such. Theimages stored on a PACS will be the same as they were viewed by theradiologist or other health professional during the detection/diagnosisor other earlier review. Another advantage of the system of FIG. 2 isthe reduced system reconstruction burden compared to the system in FIG.1, where the reconstruction engine is just prior to the display. Ifthere are multiple acquisition systems, for example four systems thatare all pushing images to the display, then the reconstruction enginewill need to reconstruct images at 4 times the rate of a reconstructionengine in a system having only one acquisition system, for the sametotal patient throughput.

An example of such a four-acquisition station system using the design ofFIG. 1 is illustrated in FIG. 3. An example of a four-acquisitionstation system using the design of FIG. 2 is illustrated in FIG. 4, andthis system can reconstruct more images in a given amount of time due tothe increased number of reconstruction engines.

The question of which system design will place a greater burden on thePACS storage of an institution will depend upon the sizes of the rawprojections Tp and of the reconstructed images Tr. In general, if theraw projections Tp are smaller than the reconstructed images Tr, itmight be desirable to save in PACS the raw or preliminarily processeddata for Tp images and reconstruct the final Tr images on demand fordisplay or other use. In all cases it may be desirable to keep both ofthese sizes as small as possible.

One way to reduce the size of an original dataset for a Tp image is tobin the projection Tp data to as large a pixel size as practical withoutreducing clinical efficacy of the final Tp or Tr images. It can beparticularly useful to bin the pixel data asymmetrically, with a coarserbin in the direction of motion of a source of the imaging x-raysrelative to the breast being imaged and a finer bin in the orthogonaldirection, as described in at least one of said earlier-filed patentapplications. The binning can be done as a part of the X-ray dataacquisition process, in the course of reading out measurement data froma flat panel digital imager. Alternatively, it can be done after initialdata acquisition. Compression of the projections using lossless or lossycompression algorithms can also serve to reduce the image size. Thereare different known ways to reduce the size of the reconstructeddatasets such as those for Tr images, and this can be particularlyimportant if the reconstructions are being saved in PACS, and if theyare being transmitted through the hospital or other facility network.Data compression is one way to reduce dataset size. Another is to makethe reconstructed pixel sizes as large as practical consistent with theclinical imaging task. It is believed that, as one non-limiting example,a pixel size of 140 microns×140 microns for the reconstructed slices isreasonable for many if not most viewing purposes. The display system caninterpolate or extrapolate along either or both the image directions toa finer pixel size for display, and this can be useful when it isdesired to conform to the pixel size of another image, such as a digitalmammogram taken at a finer resolution than 140 microns. It is alsofaster to reconstruct into a coarser pixel size in either or both imagedirections and then perform display interpolation or extrapolation to afiner pixel size, and doing so may not affect clinical efficacy as longas the reconstructed pixel size is adequately fine.

The tomosynthesis acquisition system can be dedicated to Tp imageacquisition, or it can be capable of acquiring either mammograms Mp ortomosynthesis images Tp (reconstructed into tomosynthesis images Tr) inseparate imaging sessions, or both in a single imaging session, asdescribed in said earlier-filed applications. Thus, a display systempreferably should be able to display both Mp and Tr (and/or Tp) imagesconcurrently or sequentially or in toggled mode. Similarly, the displaysystem preferably should be able to display the current images as wellas additional images taken at other times. The tomosynthesis acquisitioncan acquire mammograms and tomosynthesis images Tp in a singlecompression, as described in said earlier-filed applications. In such acase, because the breast geometry is essentially unchanged between thetwo image types, a location in an Mp or Tr image can be related to thesame breast location in the other image. If correlative geometry exists,the two image types can be overlaid on top of each other, and the usercan toggle back and forth between which image type is visible at a giventime. Thus, in general, the display can simultaneously or sequentiallyor in toggled mode display mammograms and tomosynthesis images Tr(and/or Tp) from the current and previous studies. Tr slices can bereconstructed all to the same size, which can be the same as the size ofan Mp image of the breast, or they can be initially reconstructed tosizes determined by the fan shape of the x-ray beam used in theacquisition and later converted to that same size by appropriateinterpolation/extrapolation.

Methods of identifying which image corresponds to which image type indisplays of Mp, Tr and/or Tp images are desirable. One example of such amethod is illustrated in FIG. 5. An icon is used to identify an imagetype. In this non-limiting example, the symbol M on the left imageindicates that it is a mammogram. The symbol T on the right imageindicates that it is a tomosynthesis slice image Tr. Similarly, a symbolTp (not shown) can be used to indicate that the displayed image is atomosynthesis projection image Tp, and the symbol 3D (also not shown)can be used to indicate that an image on the display is a 3D image.Other symbols/icons serving a similar purpose can be used instead of, orin addition, to those identified above. In the alternative, the imagescan be displayed without an identification of the type of image. Forexample, a Tr image and an Mp image can be displayed at the same time ortoggled without displaying an indication of the type (Tr or Mp) of theimage that is visible. This may be desirable in cases such as when auser has a familiar hanging protocol and does not need an expressidentification of the type of image.

The system described as a non-limiting example in this patentspecification is capable of receiving and displaying selectively thetomosynthesis projection images Tp, the tomosynthesis reconstructionimages Tr, and/or the mammogram images Mp, or a single type, or anysubcombination of types. It can receive images stored uncompressed,losslessly compressed, and lossyly compressed. It can also includealgorithms to decompress images sent in compressed format. The systemhas software to perform reconstruction of tomosynthesis image data forimages Tp into images Tr. Further, it can include software to generate3D display images from the tomosynthesis reconstructed images Tr usingstandard known methods such as MIP (Maximum Intensity Projection),summing, and/or weighted summing algorithms.

Referring to the tomosynthesis image Tr displayed on the right of FIG.5, a slider bar indicates by means of a short horizontal bar the heightof the displayed slice, in this example above the breast platform,although the height could be related to other references instead. Inthis case the height is approximately 5 cm, as seen from the marks 0, 2,4, 6 (cm) on the vertical bar. The height of a Tr slice that isdisplayed can be changed using a standard computer interface, such as akeyboard or mouse or mouse wheel or trackball. When the height changes,the slider bar updates by moving up or down to accurately reflect thedisplayed slice. Another method of display is an overlay method, wherethe mammogram Mp and the tomographic slice image(s) Tr are stacked oneon top of another. This is illustrated in FIG. 6. The symbol TM in thisnon-limiting example means that the display is an overlay of at leastone tomosynthesis image Tr plus a mammogram image Mp. The visible image,that is the image type on top, can be changed from Tr to Mp and viceversa easily, such as toggling back and forth using a keyboard oranother interface device. The image type that is visible can beidentified by changing the symbols such as bolding or underlining thetop one. For example, if the image Tr was on top, the symbol could be TM(T is foldface), while if the image Mp was on top the symbol could be TM(M is boldface). Alternatively, TM can be used when the Tr image isvisible and MT when the Mp image is visible, or some other way can beused to show which image is on top and which is on the bottom of thedisplay stack. As noted above, the top image can be made partlytransparent, and other techniques such as fading one image into theother can be used. FIG. 6 further illustrates another display method. Inaddition to the slice height of a Tr image, the slice thickness can beadjusted and displayed, preferably non-numerically. In this example, thethickness of the horizontal bar that is between the marks for 5 cm and 6cm indicates slice thickness scaled to the cm marks on the verticalscale. Alternatively, the displayed slice height and/or thickness can bedisplayed in a numeric format. Typically, the breast slices representedby Tr images are thin, on the order of 0.5-3 mm, and will not showobjects that are far from the given slice. If it is desired to viewobjects seen from a thicker slice, one can perform reconstructions togenerate Tr images of synthesized thicker slices, such as 5, 10, 15 or20 mm or more, or two or more Tr images can be blended into a single Trimage representing a thicker slice. The blending can be with the same ordifferent weighting of the original Tr images.

The selection of which Tr slice image should be displayed can be handledin a number of different ways. The user can click or drag the horizontalslider bar to the desired slice height, and the display would follow.Alternatively, the height could be selected using keyboard commands,mouse wheels or trackballs, or other such computer selection tools. TheTr slice image can be played in cine mode, with the speed and directioncontrollable by the user. Tomosynthesis projection images Tp can also bedisplayed in cine mode. If two sets of Tr images are displayed at thesame time, for example a set from a current acquisition and a set fromlast year's acquisition for the same breast, it is possible that thethickness of the compressed breast changed, for example because theperson's weight changed of because different compression was used. Insuch case, if each Tr image represents a breast slice of equalthickness, the number of images in one set may not be the same as thatin the other. Several options are available for displaying the two setsconcurrently in a scroll or cine mode. One is to move through the twosets at the same speed, in which case the end image of one set mayremain on the screen while images from other set still change. Anotheris to move through the images at different speeds such that the endimages of the two sets appear on the screen at the same time. Yetanother is to change the images of one set (or both sets) such that thenumber of Tr images in each set is the same. This may involveinterpolation/extrapolation that effectively changes the thickness ofthe slices in one or both sets, or omission of slices or repetition ofthe display of one or more slices. Similar synchronization or displaymode selection is provided between scroll/cine displays of images fortwo breasts,

As noted above, if two or more sets of tomosynthesis images aredisplayed concurrently, for example Tr images from the same breast takenat different times, or Tr images of the two breasts taken at the sametime, these two sets of images can be simultaneously displayed in cinémode. The ciné displays can be synchronized, so that if these twodatasets represent the same breast, the ciné display of both willtraverse through each breast dataset at the same rate.

The display of the slice image Tr has, in addition to the display of theslice height, a graphical method of displaying the corresponding slicethickness. The width of the cross-bar shown in FIG. 6 illustrates theslice thickness.

These Tr images for thicker slices can be derived in several ways. Oneway is to sum together a number of the adjacent thinner Tr slice images.Another is to calculate a maximum intensity projection through theadjacent slices. Yet another way to change the slice thickness is toreconstruct the dataset using a subset of the projections Tp. If oneuses fewer projections, this is equivalent to an acquisition over ashallower angle and consequently the reconstructed images Tr have agreater depth of field and thus represent thicker slices. For example,if only one projection is used to reconstruct, this represents atomosynthesis acquisition over a 0° angular scan and the depth of fieldis infinite, i.e. the reconstructions are 2D, as in an Mp image of thesame breast.

In the most general case, the display screens will contain a mixture ofmammogram Mp, tomosynthesis Tr and/or Tp, and combination (Mp+Tp/Tr)images. One example of this is illustrated in FIG. 7. It shows a 4-viewexamination being compared to a prior 4-view exam, where different viewsof different breasts are either Mp, Tr/Tp, or combination displays. Thesoftware allows the selection of one or more image planes, for use inimage processing, or to change window/level or to change slice height,etc. The selected image planes are indicated in this case in some way;in this non-limiting example the selected plane is outlined with adotted line. These sets of images can be on one monitor, or on multiplemonitors or other displays. In the alternative, one or more or all ofthe displayed images can be shown without an identification of the typeof image, e.g., without notation such as TM or M

When more than one image is displayed, it can be convenient to have theimages all be displayed at the same pixel spacing, using knowninterpolation or extrapolation methods applied to digital images. Thiscan facilitate image comparison. As an example, if the prior mammogramwas acquired on a system using 100 micron pixel spacing, but the currentmammogram was acquired on a system using 70 micron pixel spacing, thedisplay can map the images so the pixel spacings are identical. Thispixel spacing adjustment can also be used for Mp and Tr/Tp images. In apreferred embodiment, the Mp and Tr/Tp images are displayed at the samepixel size. This is especially useful in performing overlaid or toggledimage display, with the Mp and Tr/Tp images on top of each other. Thus,an object in a Tr image will appear at the same place as in thecorresponding Mp image. If the two images arc not at the same pixelsize, toggling between them may show a distracting change due to thedifference in pixel size. Matching the pixel spacings for all images onthe display is only one possibility. A capability to change the pixelspacings of any image or sets of images, such as would occur when onezoomed a region of a breast, can also be included.

Zooming can be done on any of the images on the display. For example, ina combo overlay display mode, the zoomed area will zoom both the Mp andthe Tr slice images as they are toggled. In other words, no matter whatimage type is displayed, it will be zoomed. Window/level can beindependently, or jointly, applied to any combination of images on thedisplay. In particular, for the Tr images the window/level can beapplied to just the single displayed Tr slice image, or all the Tr sliceimages. If there is a magnified region of an image, window/level can beselectively applied just to the magnified region or to the entire image.

A compressed breast is frequently about 50 mm thick, and if thereconstructed slice pitch or separation is 1 mm then the examinationwill consist of 50 slices. The time it takes to load this study intodisplay might be significant. Because of this, it can be useful for thedisplay to indicate the status of the display if the image is currentlybeing loaded. This can take the form of a message such as “imageloading” or an icon indicating the same, or information providing moredetail regarding loading status such as, without limitation, remainingtime for completed display.

The sequence of displaying Tr images can be controlled to select eitherthe first, last, or middle, or some other Tr slice image, as the initialslice to display. This control also defines the starting slice thicknessto display.

CAD algorithms are commonly used to analyze mammograms. CAD can also beapplied to Tr and/or Tr images. It can be useful to display CAD marksthat are derived from or are otherwise associated with the Tr/Tp images,at or for the appropriate locations on the Mp images. For example, whena Tr slice image is displayed that contains one or more CAD marks, thex,y location of the CAD mark on the Tr slice image is used to computethe corresponding x,y location on the Mp image that represents the samebreast location. The mark can then be placed on one or both of the Mpand Tr images at the same locations. Similarly, it can be useful todisplay CAD marks that are derived from or are otherwise associated withthe Mp images on the appropriate locations on the Tr slice images. Forexample, the x,y location from the Mp CAD mark is used to compute thecorresponding x,y location on the Tr slice image that represents thesame breast location, and the mark is placed on the Tr slice.

One method of displaying CAD information is illustrated in FIG. 8. Slicelocations where there are CAD marks are indicated. In this example, theyare indicated though the use of arrows positioned at the slice heightswhere the marks are. In this non-limiting example, there were CAD marksat heights 1 and 3 cm, and the currently displayed slice is at 5 cm.

Another display method for use with Tr images that have Tr CAD data isto restrict the display of Tr slice images that do not have CAD marks onthem. For example, if only Tr slice images 10 and 20 had CAD marks, thenonly those two slice images would be displayed. This allows the speedupof image review, because there can be 50 or more Tr slices that need tobe displayed. The image display could jump from one CAD-marked sliceimage Tr to another quickly. There can also be an override method sothat all the slice images Tr could be reviewed if desired.

In addition to CAD information display, the unit can display patientdemographic and acquisition information relevant to the acquisition andreconstruction of Tp/Tr images.

There also are different methods of printing Tr/Tp images. Because thereare many slice images Tr, it may not be desirable to print out eachindividual slice image on separate sheets. In this case, the system cansupport printing of the Tr images in an N×M film layout format. Inaddition, printing can be allowed in a screen capture WYSIWYG (What YouSee Is What You Get) format, or for only selected ones of the images,for example images that have CAD marks associated with them.

A common method of reviewing digital mammography and tomosynthesisimages Mp, Tp, and Tr is by using one or more monitors, and looking atthe images in an essentially monoscopic mode-the same image is viewed byboth of the viewer's eyes. Researchers have proposed using stereoviewing systems, whereby different images are presented to the left andright eyes. This method of viewing is known as stereoscopic, and canoffer distance or depth cues similarly to what is normally seen by humaneyes in regular vision tasks. Stereoscopic viewing offers potentialbenefits in viewing radiological images, because relative spatialrelationships between objects in the body might be more apparent. Onesuch stereoscopic system, for use in medical displays, is proposed inU.S. Pat. No. 6,031,565 issued on Feb. 29, 2000 and involves taking tworadiographic images of a body from different angles. The display ofthese two images provides depth information.

Tomosynthesis images offer new opportunities for improved stereoscopicviewing, at least in part because it provides a richer dataset than justa stereo pair to be displayed, it provides many possible combinations ofimage pairs, and provides for scrolling through different displayed setsof images.

One method of display using pairs of images from the tomographicprojection dataset Tp is illustrates in FIG. 9. Any two pairs ofprojections Tp may provide stereo visualization, and by displayingdynamically sets of these pairs of projections, one will get both astereoscopic view and one which dynamically moves around the body thatwas imaged. For example, consider that 8 projections Tp were taken as apart of a tomosynthesis acquisition: Tp[1], Tp[2], . . . Tp[21]. Thefirst pair to be stereo viewed could be Tp[1] and Tp[3], the second pairTp[2] and Tp[4], the third pair Tp[3] and Tp[5] and so on to Tp[6] andTp[8]. Alternatively pairs could be adjacent pairs such as Tp[1] andTp[2] or separated by three projections Tp[1] and Tp[4], etc. Theoptimal spacing between the two projections in the displayed stereo pairis dependent upon the imaging geometry and angular separation betweensuccessive projections Tp[i] and Tp[i+1]. It is known that only certainangular differences between stereo pairs give good stereo visualizationto humans, and the selection of appropriate pairs of images Tp for aparticular acquisition setting can be determined through convenientexperimentation.

Another method of display uses a variant of the reconstructed dataset Trand is illustrated in FIG. 10. In this embodiment, one reconstructs twodifferent datasets Tr′ and Tr″, both reconstructed from some or all ofthe original projections Tp. When performing reconstruction, one choosesthe geometry of reconstruction and it is possible to reconstruct thesetwo datasets into images Tr that differ in their view of the body by aselected angular separation, e.g. a few degrees, thus mimicking theapparent viewing of the body that the human eye would see, if it hadx-ray vision. Each dataset Tr′ and Tr″ consist of sets of crosssectional slices. If one displays, using the stereo viewer, the sameslice from Tr′ and Tr″, one in the left eye and the other in the righteye, one would get a stereo perspective of that slice. One could displaydifferent slice pairs in depth succession in the breast to get a dynamicstereo view through the body. As seen in FIG. 9, an object 3000 isreconstructed with cross-sectional slices Tr′ and Tr″ perpendicular totwo different axes 3001 and 3002. The slice pairs that are displayed tothe left and right eye are Tr′[i] (3003) and Tr″[i] (3004). The anglebetween 3001 and 3002 is such that viewing the pairs Tr′ and Tr″ gives astereoscopic visualization. Unlike the proposal in said U.S. Pat. No.6,031,565, FIG. 9 illustrates a stereo view of a thin cross-sectionalslice through the body, and a scroll through such sections, while thepatent proposes displaying stereo pairs of projection radiographsthrough the entire body.

FIG. 11 illustrates an overall mammography/tomography system in whichthe preferred but non-limiting examples discussed above can beimplemented. The Figure illustrates in block diagram form an x-ray dataacquisition unit 100 that includes an x-ray source 110 imaging a breast112 supported on a breast platform 114. An x-ray imager 116 such as aflat panel x-ray imager commercially available from the assignee of thispatent specification generates projection image data that can be amammogram Mp or a tomosynthesis projection image Tp. X-ray source 110 ismounted for movement around breast platform 114 so that images Tp can betaken at different angles. X-ray imager 116 can be stationary or it canalso move relative to breast platform 114, preferably in synchronismwith movement of x-ray source 110. Elements 110 and 116 communicate withx-ray data acquisition control 118 that controls operations in a mannerknown from said earlier-filed patent specifications. X-ray image datafrom imager 116 is delivered to processing and image reconstruction unit120, where the data is processed as known from said earlier-filed patentapplication into Tp and Tr image data, possibly stored, and prepared fordisplay at image display unit 122 as disclosed in the variousembodiments described above. The appropriate software to carry out theprocesses describe above can be written by programmers skilled in theart based on the disclosure above and general knowledge in the artwithout due experimentation, and in general will be different fordifferent data processing platforms.

FIG. 12 illustrates a non-numeric way of indicating the orientation ofbreast slices that are represented by Tp images. In this example, thethicker vertical line indicates a direction normal to the breastplatform on which the compressed breast rests for the Tp image and thethinner line indicates the orientation of a slice represented by a Tpimage (not shown in FIG. 12) that is being displayed.

The examples described above are only illustrative and that otherexamples also are encompassed within the scope of the appended claims.It should also be clear that, as noted above, techniques from knownimage processing and display methods such as post-production of TVimages and picture manipulation by software such as Photoshop fromAdobe, can be used to implement details of the processes describedabove. The above specific embodiments are illustrative, and manyvariations can be introduced on these embodiments without departing fromthe spirit of the disclosure or from the scope of the appended claims.For example, elements and/or features of different illustrativeembodiments may be combined with each other and/or substituted for eachother within the scope of this disclosure anHd appended claims.

The invention claimed is:
 1. A method of displaying tomosynthesisimages, the method comprising: obtaining a first set of tomosynthesisimages, wherein the first set of tomosynthesis images includes a firstnumber of images; obtaining a second set of tomosynthesis images,wherein the second set of tomosynthesis images includes a second numberof images different than the first number of images; and modifying thefirst set of tomosynthesis images so as to produce a modified first setof tomosynthesis images having a first number of images equal to thesecond number of images.
 2. The method of claim 1, wherein the modifyingoperation includes omitting at least one image from the first set oftomosynthesis images.
 3. The method of claim 1, wherein the modifyingoperation includes duplicating at least one image of the first set oftomosynthesis images.
 4. The method of claim 1, wherein the modifyingoperation includes combining at least two images of the first set oftomosynthesis images.
 5. The method of claim 1, further comprisingdisplaying the modified first set of tomosynthesis images and the secondset of tomosynthesis images.
 6. The method of claim 5, wherein thedisplaying operation includes synchronizing display of the modifiedfirst set of tomosynthesis images and the second set of tomosynthesisimages.
 7. The method of claim 1, wherein at least one of the obtainingoperations includes accessing a storage medium.
 8. The method of claim1, wherein at least one of the obtaining operations includes imaging acompressed breast.
 9. A method of displaying tomosynthesis images, themethod comprising: acquiring a first set of tomosynthesis images,wherein the first set of tomosynthesis images includes at least one of afirst set of tomosynthesis projection images Tp and a first set oftomosynthesis reconstruction images Tr; preparing the first set oftomosynthesis images; obtaining from a storage device a second set oftomosynthesis images, wherein the second set of tomosynthesis imagesincludes at least one of a second set of tomosynthesis projection imagesTp and a second set of tomosynthesis reconstruction images Tr; andpreparing the second set of tomosynthesis images.
 10. The method ofclaim 9, wherein the first set of tomosynthesis images is the first setof tomosynthesis projection images Tp, and wherein the preparingoperation includes reconstructing the first set of tomosynthesisreconstruction images Tr from the first set of tomosynthesis projectionimages Tp.
 11. The method of claim 10, wherein the second set oftomosynthesis images is the second set of tomosynthesis projectionimages Tp, and wherein the preparing operation includes reconstructingthe second set of tomosynthesis reconstruction images Tr from the secondset of tomosynthesis projection images Tp.
 12. The method of claim 9,wherein the first set of tomosynthesis images includes a first number ofimages and wherein the second set of tomosynthesis images includes asecond number of tomosynthesis images different than the first number,and wherein the preparing operation includes changing at least one ofthe first set of tomosynthesis images and the second set oftomosynthesis images such that both the first set of tomosynthesisimages and the second set of tomosynthesis images include an equalnumber of the first set of tomosynthesis images and the second set oftomosynthesis images.
 13. The method of claim 12, further comprisingdisplaying the equal number of the first set of tomosynthesis images andthe second set of tomosynthesis images.
 14. The method of claim 13,wherein the changing operation includes omitting at least one image ofat least one of the first set of tomosynthesis images and the second setof tomosynthesis images.
 15. The method of claim 13, wherein thechanging operation includes duplicating at least one image of at leastone of the first set of tomosynthesis images and the second set oftomosynthesis images.
 16. The method of claim 13, wherein the changingoperation includes combining at least two images of at least one of (1)the first set of tomosynthesis images and (2) the second set oftomosynthesis images.
 17. The method of claim 13, wherein the displayingoperating includes synchronizing display of the equal number of thefirst set of tomosynthesis images and the second set of tomosynthesisimages.
 18. The method of claim 12, further comprising displaying afirst image of the first set of tomosynthesis images.
 19. The method ofclaim 18, further comprising: displaying an information bar on adisplay; and displaying a location bar on the information bar, wherein aposition of the location bar on the information bar corresponds to afirst location within the compressed breast of the first image.
 20. Themethod of claim 19, further comprising: receiving an input to display asecond image of the plurality of the first set of tomosynthesis images;displaying the second image; and displaying the location bar on theinformation bar in a second position corresponding to a second locationwithin the compressed breast of the second image.